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(54) Remote phone-based access to a universal multimedia mailbox 



(57) A system that provides users with the capability 
to access messages contained in a remote universal 
multimedia mailbox interfaces with an underlying uni- 
versal messaging server which provides a repository for 
voice, fax, e-mail. etc.... This interlace is accomplished 
through a standard e-mail protocol used to add voice 
reply messages from a message generator to the uni- 
versal mailbox through the universal messaging server 



as well as through a message management library con- 
tained within a mailbox access manager that provides 
access to message components contained in the uni- 
versal mailbox. The operation of the present invention 
is guided by a telephony attendant which interfaces with 
a telephone. The telephony attendant also controls a 
media converter which converts media to fax or media 
to audio and an audio/lax player which interfaces with 
the telephone or a tax machine. 
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Description 

This patent application is related to US patent application serial number 08/815,786, entitled "System For Digital 
Video Browsing Using Telephone And Fax" assigned to the same assignee as the present invention. 
s The present invention relates to accessing messages contained in remote universal multimedia mailboxes and 

more particularly to extending access to environments where computer systems are not available. 

Electronic messaging services including electronic mail, voice mail and digital fax are common in today's business 
and home environments. In addition a trend towards universal or unified messaging is currently underway. The present 
invention develops a system that operates under the assumption that an extensible Universal Messaging System with 
10 basic message storage-and-f orward capabilities is in place. Such systems are now commercially available as described 
in "Unified Messaging" Computer Telephony Magazine, August 1996. 

The prior art also includes voice mail systems such as the one described in U.S. Patent Number 5,568,540 entitled 
"Method and Apparatus for Selecting and Playing a Voice Mail Message". This patent discloses a system which provides 
telephone-based remote message access. These systems however, are limited to voice and, in some instances, fax. 
15 The present invention seeks to provide a system that addresses much more elaborate messages containing arbitrarily 
complex multimedia objects. 

Some of the systems described in "Unified Messaging" also provide support for remote access of e-mail messages. 
However, this support is based on text-to-speech conversion only and is limited to the text ponion of the message 
body. The present invention seeks to extend this functionality to every message class in the system, and to enhance 
20 \\ by incorporating the use of fax as away to provide visual feedback on the content of the message. For example, an 
image or a video could be provided. 

The present invention also seeks to provide a system capable of handling two special message classes: Dynamic 
image annotation which involves computer-generated messages in which synchronized voice, mouse movements and 
graphics are used to describe conditions associated with the contents of an underlying image; and brpwsable video 
2S messages which consists of video clips which have been pre-processed to identify scene changes and prepared for 
'browsing. Such message types are not available in existing prior art systems. 

The invention provides a system and a method for remote access to a universal multimedia mailbox, as set out in 
claims 1 and 11. 

The present invention provides users with the capability to access messages contained in a remote universal 
30 multimedia mailbox. The system of the present invention interfaces with an underlying universal messaging server 
which provides a repository for voice, fax, e-mail, etc., by means of a message management library contained within 
a mailbox access manager of the present invention that provides access to message components contained in the 
universal mailbox. The operation of the present invention is guided by a telephony attendant of the present invention 
which interfaces with a telephone. The telephony attendant also controls a media converter which converts media to 
35 fax or media to audio and an audio/fax player which interfaces with the telephone or a fax machine. The telephony 
attendant also communicates with the mailbox access manager to retrieve the number of messages in the mailbox. 
Overall, the present invention acts as a remote phone/fax messaging client (RPFMC). Advantageously, a standard e- 
mail protocol is also used to add voice reply messages from a message generator of the present invention to a universal 
mailbox through a universal messaging server. 
40 For a better understanding of the present invention, and to show how it may be brought into effect, reference will 

now be made, by way of example, to the accompanying drawings, in which: 

Figure 1 illustrates a block diagram of the present invention and its interface to a universal mailbox, a universal 
messaging server, message sources and a telephone network. 

Figure 2 illustrates a flow diagram of one embodiment of the telephony attendant of the present invention showing 
45 the global flow of a call. 

Figure 3 illustrates a flow diagram of one embodiment of the mailbox access manager of the present invention 
where the mailbox status prompt is generated and played to the caller. 

Figure 4 illustrates a flow diagram of the operation of the present invention when checking whether a fax message 
has been prepared, when prompting the user for the necessary information, and when sending the fax to the destination 
50 fax machine. 

Figure 5 illustrates a flow diagram of the telephony attendant of Figure 2 communicating with the mailbox access 
manager of Figure 3 to generate the mailbox status fax. 
Figure 6 illustrates a typical mailbox status fax. 

Figure 7 illustrates a flow diagram of the loop that controls the operation of providing a more detailed view of the 
55 messages contained in the mailbox. 

Figure 8 illustrates a flow diagram of the steps used to review the contents of a fax. 

Figure 9 illustrates a flow diagram of the steps used to review the contents of a voice message. 

Figure 10 illustrates an example of the results produced by the media converter of the present invention in response 
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to a request tor phone/fax access to an annotation. 

Figure 11 illustrates a flow diagram of the operation of the telephony attendant of the present invention during an 
annotation review process. 

Figure 12 illustrates an example of a video fax of the present invention. 
5 Figure 13 illustrates a flow diagram of the operation of the telephony attendant of the present invention during a 

video review process. ,i 

Figure 14 illustrates a flow diagram of the operation of the telephony attendant of the present invention during an 

E-Mail review process. 

Figure 15 illustrates an example of a telefax transmittal of the present invention describing the contents of an e- 
io mail message. 

Figure 16 illustrates a flow diagram of the operation of reviewing individual attachments. 

An exemplary embodiment of the invention will now be described, with reference to the accompanying drawings. 
The present invention is a system that provides the capability for people to access messages contained in a remote 
universal multimedia mailbox. Typically, the universal mailbox contains a combination of the following message classes: 
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• voice mail: voice only messages typically generated via a phone 

• tax: document images generated with a lax machine or a fax program 

multimedia e-mail: collection of multimedia objects (text, voice, images,...) structured according to an accepted 
standard, e.g., MIME 

20 • dynamically annotated images: computer generated messages containing synchronized voice, graphics and 
mouse gestures used to describe conditions associated with an underlying image. 

• video: video messages generated in a video phone or a workstation 

Under normal circumstances this universal mailbox is accessed via tools that run on a fully functional computer 

25 system such as a personal computer or workstation. The present invention extends access to these message classes 
in environments where computer systems are not available, but telephones and fax machines are. 

The operation of the present invention in a basic scenario is as follows. A user will call a predefined telephone 
number at the site where the mailbox is stored. The call is answered and processed at this site by an automated 
telephony attendant. This attendant gives users verbal instructions on how to access the messages and receives user 

30 commands in the form of touch tones generated with the keypad of a telephone. The system responds to user requests 
by playing audio versions of messages or message components over the phone, and visual representations by means 
of a fax machine which is presumed to be close to the place where the telephone call originates. A series of unique 
media conversion mechanisms is used together with off-the-shelf tools to support the desired functionality. 

A particular feature of the present invention is the systems ability to handle two special message classes: Dynamic 

35 image annotation which involves computer-generated messages in which synchronized voice, mouse movements and 
graphics are used to describe conditions associated with the contents ol an underlying image; and browsable video 
messages which consists of video clips which have been pre-processed to identify scene changes and prepared for 
browsing. Such message types are not available in existing prior art systems but are incorporated into the underlying 
Universal Messaging Server using the techniques described in UTSTPaterit Application Serial Number 08/560.566, 

40 filed November 20, 1995, entitled "A Multimedia-based Reporting System With Recording And Playback Of Dynamic 
Annotation" and assigned to the same assignee as the present invention as well as U.S. Patent Number 5,521,841 
entitled "Browsing Contents Of A Given Video Sequence", issued May 28, 1996 and assigned to the same assignee 
as the present invention. 

Figure 1 illustrates the main components of the present invention as well as the flow of data and messages. The 
45 present invention assumes that an underlying Universal Messaging Server, e.g., Microsoft's Exchange Server, will 
provide a single repository for voice mail 11 , tax 12, and e-mail 13 which are provided from message sources 15. Other 
than receiving messages from message sources 15, universal messaging server 21 also delivers messages out to 
their destinations. In addition, the system is presumed to be extensible through server extensions 16 in order tosupport 
the incorporation of the dynamic annotated images 17 and video classes 18 
so Server extensions 1 6 allow new message classes to be represented customarily. Server extensions 16 also facil- 

itates the pre-processing of those messages. For example, a video message could be passed to a scene change 
detector which generates the scene change icons and scene change list. The dynamic annotation inside an annotated 
image message could be segmented with reference to time and spatial information such that a user could jump into a 
particular segment of annotation during playback of the message. 
55 As shown in Figure 1, there are two interlaces between the present invention 10 and the universal messaging 

server 21 : (1 ) the standard e-mail protocol used to add voice reply messages 22 1rom message generator 23 to universal 
mailbox 24 through universal messaging server 21; and (2) a message management library contained within mailbox 
access manager 25 that provides access to message components 26 contained in the universal mailbox 24. These 
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are typical interfaces ol existing messaging servers. Overall the present invention acts as a remote phone/fax mes- 
saging client (RPFMC). 

The operation. of the RPFMC 10 is guided by a telephony attendant 27 that performs the following functions: 

• Receives the call 2B from a telephone 29, greets the caller, and prompts 30 the customer t for identification. 

• Guides a session during which the customer reviews the contents of the mailbox. 

• Issues queries 40 to the mailbox access manager 25 to retrieve message components 26 from mailbox 24, and 
interacts with media converter 32, which receives media message components 33 from mailbox access manager 
25 and performs the transformations needed to present the datavia phone 29 or fax 34. 

• Delivers audio and fax data 36 to audio/fax player 38 which delivers the information as audio data 35 through 
phone 29 or as fax data 37 through fax 34. 

• Records voice replies to e-mail messages 41 and handles them through message generator 23. 

More specifically, telephony attendant 27 performs the following tasks as illustrated in Figure 2. 

• Answer the call, play a greeting prompt 30, and wait for the user to enter a password. 

• It the password is not correct, inform the user by playing an "Unauthorized User" prompt and end the call playing 
a "Dismissal" prompt. 

• If the password is correct, report the status of the mailbox (Figure 3) and provide a menu of alternatives as described 
in the mailbox-review prompt. 

• Wait for the selection and perform the function selected , 

Playing the status prompt involves interaction through message query 40 between telephony attendant 27 and 
mailbox access manager 25, which in this situation examines the contents of mailbox 24 and returns a count of the 
messages found. This count is used to create a dynamic prompt 30 which is played to the user through the phone 29. 
This is further illustrated in the flow diagram of Figure 3. 1 

The user can reply to the originator of e-mail, through reply data 41, with the message generator 23. Message 
generator 23 records voice from the user, composes it as an E-mail message and submits it to universal messaging 
server 21 which delivers it as a voice message to the originator. \ 

The telephony attendant follows the playback of the status prompt, with the playback of a prompt that offers three 
options to the caller: 

• retrieve a high-level description of the mailbox (mailbox status fax) via fax: • 

• proceed to review each message in detail; or, 

• exit the system. 

In the last case the system of the present invention first checks to see if a fax message has been prepared and 
prompts the user for the necessary information. 

Figure 4 illustrates a flow diagram of the operation ol the system in this case. 
— During the-generation of the mailbox status fax, the telephony attendant communicates with the mailbox access 
manager to retrieve the number of messages in the mailbox. A flow diagram of this operation is illustrated in Figure 5. 
The telephony attendant then proceeds to systematically add information about each individual message to an internal 
fax document. This information includes: 

• the message type, 

• the sender, 

• the subject, 

• the dale, and 

• the number of attachments (e-mail only). 

Figure 6 illustrates a typical mailbox status fax. A document like this can be created by using a programmable 
word processing tool such as Microsoft Word 6.0, 

The next option available in the mailbox review prompt is a more detailed view of the messages contained in the 
mailbox. This option is made available to the user by means of a dynamically generated prompt that systematically 
plays back audible high-level message components (message type, sender, subject, etc. ...). Figure 7 illustrates a flow 
diagram of the loop that controls this operation. Notice the interaction with the mailbox access manager to extract 
message information from the mailbox. In particular, the message type which determines the way in which the message 
is reviewed. 
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Figure 8 illustrates a flow diagram for reviewing the contents of a fax and Figure 9 illustrates a flow diagram for 
reviewing the contents of a voice message. [ 

The services of media converter, 32 of Figure 1, are used to review the contents of dynamic image annotations 
and video messages both of which have aural and visual representations. For dynamic annotations, U.S. Patent Ap- 
plication Serial Number 08/560,566 filed November 20, 1995, entitled "A Multimedia-based Reporting System With 
Recording And Playback Of Dynamic Annotation", assigned to the same assignee as the present invention, teaches 
the use of three separate components in the storage organization of the multimedia object: an audio file with the voice 
comments, an image file with the bitmap(s), and a text file with gestures and graphics. Similarly, U.S. Patent Number 
5,521,841 entitled "Browsing Contents Of A Given Video Sequence", assigned to the same assignee as the present 
invention, teaches an organization of the video file that includes the video itself plus auxiliary data in the form of a 
scene change list and a collection of frame icons that show the contents of the video during scene changes. 

Figure 1 0 shows a sample of the results produced by the media converter when applied to a dynamically annotated 
image. The aural component is copied directly out of the stored representation into an intermediate voice file that can 
be played over the phone. The algorithms described in , U.S. Patent Application Serial-Number 08/560,566 filed No- 
vember 20 : 1 995 are utilized to playback the dynamic annotation and are then used to produce the visual representation 
that contains the base images as well as the drawings as they would appear at the end of the annotation playback. 
Figure 11 shows a flow diagram of the operation of the telephony attendant during the annotation review process. 

The algorithms used in the media converter to decompose a video message are described in U.S. Patent Number 
5,521,841. Fiaure 12 shows a sample video fax. The audio track of the video is accessed in a telephone call during 
which' users can cortiol the playback of the audio track using the keypad. In addition to having pause/resume capa- 
bilities, the usei i=» a:t>o *b>c to jump to specific video shots by entering the numeric id associated to each shot in the 
sequence. Figure 1 3 shows a flow diagram of the operation of the telephony attendant during the video review process. 

The last component of the telephony attendant is the e-mail reviewing process. The system of the present invention 
performs the task in two steps trst giving the caller information about the main message, then allowing for the review 
of message attachments Figure 14 illustrates the flow diagram of the steps in this interaction. Once again the user is 
given the option to receive nn audible version of the message over the phone, or to receive a visual copy through the 
fax. For the audible version a commercial text-to-speech engine included in the media converter translates the mes- 
sage body. For the faxed version a telefax transmittal similar to the sample shown in Figure 15 is generated, also by 
the media converter using an off-the-shelf word processing system (e.g., MS Word 6.0). Figure 16 illustrates a flow 
diagram of the process used to review individual attachments. 

The following list contains examples of textual representations of system prompts. These prompts could be re- 
corded as voice during the implementation of the system. 



Prompt 



[11 Greeting 

121 Unauthorized User 

[3] Dismissal 

14) Mailbox Review 

151 Review Fax 

[61 Fax Saved 

[71 Get Fax Number 

[81 Fax Sent 

191 Review Voice 

110] Voice Saved 

[111 Review Annotation 

1121 Annotation Saved 
1131 Review Annotation 

[14] Annotation Saved 
[151 Review E-mail 



Content 



"Hello, you have reached the universal mailbox of <USER NAME>. Please enter your 
password." 

"The password was incorrect" 
"Good bye!" 

"To receive a summary ol the mailbox via fax press 1 . To review the messages in detail 
press 2. To exit press 3." 

"To get a copy of this fax press 1 . To save the message press 2." 
"Fax saved." 

"Please enter your fax number." 
"Fax sent." 

"To review this voice mail press 1 . To save the message press 2." 
"Voice saved." 

"To review this annotation using the phone press 1 . To receive a fax version of the visual 
components press 2. To save the message press 3." 
"Annotation saved." 

"To review this video message using the phone press 1 . To receive a fax version of the 
visual components press 2. To save the message press 3." 
"Video message saved." 

"To review this e-mail message using the phone press 1 . To receive a fax version of the 
visual components press 2. To review individual attachments press 3. To save the 
message press 4." 
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(continued) 



Prompt 


Content 


[1 61 E-mail Saved 


"E-mail saved." 



. It is not intended that this invention be limited by the hardware or software arrangement, or operational procedures 
shown disclosed. This invention includes all of the alterations and variations thereto as encompassed within the scope 
of the claims as follows. 



Claims 

1. A system (10) for remote phone-based access to a universal multimedia mailbox (24) comprising: 

mailbox access manager means (25) for interfacing with said universal multimedia mailbox: 
telephony attendant means (27) connected to said mailbox access manager means for interfacing to a tele- 
phone network; and 

media converter means (32) connected to said mailbox access manager means. 

A system (10) for remote phone-based access to a universal multimedia mailbox (24) as claimed in claim 1 further 
comprising server means (16) for allowing message classes to be represented customarily and for facilitating 
preprocessing of said message classes. 

A system (1 0) for remote phone-based access to a universal multimedia mailbox (24) as claimed in claim 2 wherein 
said server means (16) supports dynamic annotated images and video message classes. 

A system (10) for remote phone-based access to a universal multimedia mailbox (24) as claimed in one of claims 
'1-3, further comprising an audio/fax player (35, 37) connected between said media converter means (32) and said 
telephone network. 

A system (10) for remote phone-based access to a universal multimedia mailbox (24) as claimed in any preceding 
claim further comprising message generator means (23) for interfacing with said universal multimedia mailbox (24) 
through a universal messaging server (21). 

A system (10) for remote phone-based access to a universal multimedia mailbox (24) as claimed in any preceding 
claim wherein said mailbox access manager means (25) comprises a message management library that provides 
access to message components contained in said universal multimedia mailbox (24). 

A system (10) for remote phone-based access to a universal multimedia mailbox (24) as claimed in any preceding 
claim wherein said telephony attendant means (27) guides operation of said system" (1 0)1 or remote phone-based 
access to said universal multimedia mailbox (24). 

A system (10) for remote phone-based access to a universal multimedia mailbox (24) as claimed in any preceding 
claim wherein said telephony attendant means (27) interacts with said mailbox access manager means (25) which 
retrieves message components from said universal multimedia mailbox (24) and interacts through said mailbox 
access manager means (25) with said media convener means (32) which receives media message components 
from said mailbox access manager means (25) and performs transformations needed to present data to and from 
said telephone network. 

A system (10) for remote phone-based access to a universal multimedia mailbox (24) as claimed in any preceding 
claim wherein said mailbox access manager means (25) examines contents of said universal multimedia mailbox 
(24) and returns a count of messages found. 

10. A system (10) for remote phone-based access to a universal multimedia mailbox (24) as claimed in any preceding 
claim wherein said media converter means (32) reviews contents of dynamic image annotations and video mes- 
sages. 
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11. A method ot providing tor remote phone-based access to a universal multimedia mailbox comprising the steps ot: 

utilizing a mailbox access manager to interface with said universal multimedia mailbox; 
utilizing a telephony attendant; and \ , , 

utilizing a media converter. 

12. A method of providing for remote phone-based .access to a universal multimedia mailbox as claimed in claim 11, 
further comprising the steps of: 

utilizing server extensions for allowing new message classes to be represented customarily and for facilitating 
preprocessing of said new message classes; and » 
utilizing an audio/fax player for interlacing to a telephone network. 

13. A method of providing for remote phone-based access to a universal multimedia mailbox as claimed in one of 
claims 11 or 1 2 wherein utilizing a telephony attendant comprises the steps of: 

providing the capability tor users to receive instructions on how to access messages; and 
providing the capability tor receiving commands from said users. 

14. A method of providing for remote phone-based access to a universal multimedia mailbox as claimed in one ot 
" claims 11-13 further comprising the step of interfacing a message generator to the universal multimedia mailbox 

through a universal messaging server. 

15. A method of providing for remote phone-based access to a universal multimedia mailbox as claimed in one of 
claims 11-14 wherein utilizing a mailbox access manager comprises the step of utilizing a message management 
library that provides access to message components contained in said universal multimedia mailbox. 

16. A method of providing for remote phone-based access to a universal multimedia mailbox as claimed in one of 
claims 11-15 wherein utilizing a telephony attendant comprises the steps of: 

interacting with said mailbox access manager which retrieves message components from said universal mul- 
timedia mailbox: and • ,, 

interacting through said mailbox access manager means with said media converter which receives media 
message components from said mailbox access manager and performs transformations needed to present 
data to and from said telephone network. 

17. A method of providing for remote phone-based access to a universal multimedia mailbox as claimed in one of 
claims 11-16 wherein utilizing said mailbox access manager comprises the steps ol: 

examining contents of said universal multimedia mailbox; and 
returning a count of messages found. 
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(54) Remote phone-based access to a universal multimedia mailbox 



(57) A system that provides users with the capability 
to access messages contained in a remote universal 
multimedia mailbox interfaces with an underlying uni- 
versal messaging server which provides a repository for 
voice, fax, e-mail, etc.... This interface is accomplished 
through a standard e-mail protocol used to add voice 
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versal mailbox through the universal messaging server 



as well as through a message management library con- 
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to audio and an audio/fax player which interfaces with 
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